Robot swarms have been proposed for a multitude of interesting applications, with one drawback being the common assumption that swarm members are functional and cooperative. Previous literature has identified several security risks in robot swarms [1], such as the presence of intruders, system penetration, or the disruption or falsification of communicated information. Existing studies on anomalous behavior in robot swarms address various security concerns, but rely on a priori knowledge of normal or anomalous behavior patterns or focus on reaching consensus within a swarm [2, 3]. In contrast, we propose a data-driven contextual anomaly detection approach that requires only examples of normal robot motions as well as limited information about the state of the robot swarm, i.e., the robot positions. A combination of an LSTM and a normalizing flow is used to embed the state of the robot swarm and to predict a probabilistic estimate of the physical motion performed by a robot agent. Repeated deviations from its expected behavior lead to a robot being categorized as anomalous and to its subsequent exclusion from the swarm. We investigate our approach in a deployment setting, as used, e.g., in surveillance tasks [4]. Given a deployment area of variable shape and size, a robot swarm distributes within the area, seeking optimal coverage. In this setting, an antagonistic agent attempts to take control of a particular area of interest within the deployment area, e.g., with the goal of preventing the detection of illicit activities or entities. We focus on antagonistic, rather than erroneous, agents and thus implement several different behavioral strategies aimed at concealing the identity of the antagonist. For instance, robots may use out-of-distribution actions, or perform motions that only slightly deviate from normal behavior. Regardless of the strategy used, our method successfully discriminates between normal and antagonistic agents with a recall and precision of more than 80%. Similar results are obtained in hardware experiments with a robot swarm.